Local Synchronization with Batching

We examine the reduction in efficiency in a system where each processor has a sequence of tasks and must synchronize with other processors after each task is done. This occurs, for example, in many stepwise simulations. In OpenMP, the only explicit synchronization is the barrier, which causes all processors to wait for the slowest one. This is problematic when the tasks take variable amounts of time. The same inefficiency occurs when collective communication is used in MPI. However, use of send/receive commands in MPI generally reduces the synchronization to only that which is necessary, increasing efficiency. Here we look at the use of batching, whereby each processor completes several tasks before synchronizing, as yet another way to improve efficiency. We provide analytic and simulation results showing the results of batching with both barrier and send/receive synchronization models, where the task times can assume a variety of probability distributions. For the geometric and exponential distributions and cyclic dependencies, by using local synchronization and batching one can achieve any desired efficiency with a batching size independent of the number of processors. In contrast, for global synchronization and a fixed batch size, whenever the distribution has infinite support the efficiency will go to 0 as the number of processors tends to infinity.